Statistically significant differences versus convincing evidence of real treatment effects: an analysis of the false positive risk for single-centre trials in anaesthesia.

BACKGROUND: The American Statistical Association has highlighted problems with null hypothesis significance testing and outlined alternative approaches that may 'supplement or even replace P-values'. One alternative is to report the false positive risk (FPR), which quantifies the chance the null hypothesis is true when the result is statistically significant. METHODS: We reviewed single-centre, randomised trials in 10 anaesthesia journals over 6 yr where differences in a primary binary outcome were statistically significant. We calculated a Bayes factor by two methods (Gunel, Kass). From the Bayes factor we calculated the FPR for different prior beliefs for a real treatment effect. Prior beliefs were quantified by assigning pretest probabilities to the null and alternative hypotheses. RESULTS: For equal pretest probabilities of 0.5, the median (inter-quartile range [IQR]) FPR was 6% (1-22%) by the Gunel method and 6% (1-19%) by the Kass method. One in five trials had an FPR ≥20%. For trials reporting P-values 0.01-0.05, the median (IQR) FPR was 25% (16-30%) by the Gunel method and 20% (16-25%) by the Kass method. More than 90% of trials reporting P-values 0.01-0.05 required a pretest probability >0.5 to achieve an FPR of 5%. The median (IQR) difference in the FPR calculated by the two methods was 0% (0-2%). CONCLUSIONS: Our findings suggest that a substantial proportion of single-centre trials in anaesthesia reporting statistically significant differences provide limited evidence of real treatment effects, or, alternatively, required an implausibly high prior belief in a real treatment effect. CLINICAL TRIAL REGISTRATION: PROSPERO (CRD42023350783).

A single-centre, randomised controlled feasibility pilot trial comparing performance of direct laryngoscopy versus videolaryngoscopy for endotracheal intubation in surgical patients.

INTRODUCTION: Most trials comparing effectiveness of laryngoscopy technique use surrogate endpoints. Intubation success is a more appropriate endpoint for comparing effectiveness of techniques or devices. A large pragmatic clinical trial powered for intubation success has not yet been performed. METHODS: We tested the feasibility of a randomised controlled trial to compare the performance of direct laryngoscopy versus videolaryngoscopy for endotracheal intubation. The trial was conducted in the Department of Adult and Emergency Anaesthesia at the Auckland City Hospital, New Zealand. Patients over 18 years who required endotracheal intubation and were not known or predicted to be difficult to bag-mask ventilate were eligible for the study. Patients were excluded if they required rapid sequence induction, fibreoptic intubation or were unable to consent due to language barriers or cognitive impairment.Patients were permuted block randomised in groups of 8 to either direct laryngoscopy (DL) or videolaryngoscopy (VL) for the technique of endotracheal intubation. Patients were blinded to laryngoscopic technique; the duty anaesthetist, outcome assessors and statistician were unblinded.Feasibility was assessed on recruitment rate, adherence to group assignment and data completeness. Primary outcome was first-pass success rate, with secondary outcomes of time to intubation (seconds), Intubation Difficulty Score and complication rate. RESULTS: One hundred and six patients were randomised and 100 patient results were analysed. Completed data from patients randomised to the DL group (n = 49) was compared with those in the VL group (n = 51). Group adherence and data completeness were 100% and 97%, respectively. First-pass success rate was 83.7% in the direct laryngoscopy group and 72.5% in the videolaryngoscopy group (p = 0.18). Median time to intubation was significantly shorter for direct laryngoscopy when compared to videolaryngoscopy (34 s v 43 s, p = 0.038). Complications included mucosal trauma and airway bleeding which are recognised complications of endotracheal intubation. CONCLUSION: A large, pragmatic, multicentre, randomised controlled trial comparing the relative effectiveness of direct laryngoscopy and indirect videolaryngoscopy is feasible. TRIAL REGISTRATION: Australian New Zealand Clinical Trials Registry (ANZCTR), ACTRN12615001267549.